Performance driven facial animation

ABSTRACT

A method of animating a digital facial model, the method including: defining a plurality of action units; calibrating each action unit of the plurality of action units via an actor&#39;s performance; capturing first facial pose data; determining a plurality of weights, each weight of the plurality of weights uniquely corresponding to the each action unit, the plurality of weights characterizing a weighted combination of the plurality of action units, the weighted combination approximating the first facial pose data; generating a weighted activation by combining the results of applying the each weight to the each action unit; applying the weighted activation to the digital facial model; and recalibrating at least one action unit of the plurality of action units using input user adjustments to the weighted activation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of a co-pending U.S.patent application Ser. No. 12/198,762, filed Aug. 26, 2008 entitled“PERFORMANCE DRIVEN FACIAL ANIMATION”, which is a continuationapplication of a co-pending U.S. patent application Ser. No. 11/956,728,filed Dec. 14, 2007 entitled “PERFORMANCE DRIVEN FACIAL ANIMATION”,which is a continuation application of a co-pending U.S. patentapplication Ser. No. 11/739,448, filed Apr. 24, 2007, entitled“PERFORMANCE DRIVEN FACIAL ANIMATION”, which claims the benefit ofpriority of U.S. Provisional Patent Application No. 60/794,790, filedApr. 24, 2006, entitled “PERFORMANCE DRIVEN FACIAL ANIMATION.” Thedisclosures of the above-referenced patent application are herebyincorporated by reference.

BACKGROUND

1. Field of the Invention

The present invention relates generally to motion capture, and moreparticularly to methods and systems for generating facial animationusing performance data, such as motion capture data obtained from motioncapture systems and video images obtained from video data.

2. Description of the Prior Art

Modeling a face, its motion, and rendering it in a manner that appearsrealistic is a difficult problem, though progress to achieve realisticlooking faces has been made from a modeling perspective as well as arendering perspective. A greater problem is animating a digital face ina realistic and believable manner that will bear close scrutiny, whereslight flaws in the animated performance are often unacceptable. Whileadequate facial animation (stylized and realistic) can be attempted viatraditional keyframe techniques by skilled animators, it is acomplicated task that becomes especially time-consuming as the desiredresults approach realistic imagery.

Apart from keyframe techniques, other methods based on principalcomponent analysis have also been implemented to develop animated facialmodels from performance data. These methods typically generatelowest-dimensional models from the data. Further, beingmathematically-based solutions, the facial models so developed oftenlook unnatural in one or more aspects. Moreover, the resulting lowdimensionality makes post-development modification of the facial modeldifficult and non-intuitive to a user when the principal components donot correspond with natural, identifiable facial movements that can beadjusted to achieve a desired result. That is, the basis vectors(obtained using principal component analysis) do not correspond to anylogical expression subset that an artist can edit afterwards. Forexample, a simultaneous lip corner rise with eyebrow rise might besolved from performance data as single component activation. However,the single component activation may not be decoupled into separateactivations for the lip corner and eyebrow. Thus, an animator wishing toadjust only the lip corner rise may be unable to do so without alsoactivating the eyebrow component.

Therefore, what is needed is a system and method that overcomes thesesignificant problems found in the conventional systems as describedabove.

SUMMARY

The present invention provides methods and systems for generating facialanimation using performance data, such as motion capture data obtainedfrom motion capture systems and video images obtained from video data.

In one aspect, a method of animating a digital facial model isdisclosed. The method includes: defining a plurality of action units;calibrating each action unit of the plurality of action units via anactor's performance; capturing first facial pose data; determining aplurality of weights, each weight of the plurality of weights uniquelycorresponding to the each action unit, the plurality of weightscharacterizing a weighted combination of the plurality of action units,the weighted combination approximating the first facial pose data;generating a weighted activation by combining the results of applyingthe each weight to the each action unit; applying the weightedactivation to the digital facial model; and recalibrating at least oneaction unit of the plurality of action units using input useradjustments to the weighted activation.

In another aspect, a method of animating a digital facial modelincludes: defining a plurality of action units, each action unit ofincluding first facial pose data and an activation; calibrating thefirst facial pose data using calibration pose data derived from aplurality of captured calibration performances, each calibrationperformance of the plurality of captured calibration performancescorresponding with the each action unit; deriving second facial posedata from another calibration performance of the plurality of capturedcalibration performances; determining a plurality of weights, eachweight of the plurality of weights uniquely corresponding to the eachaction unit, the plurality of weights characterizing a weightedcombination of the facial pose data, the weighted combinationapproximating the second facial pose data; generating a weightedactivation by combining the results of applying the each weight to theactivation; applying the weighted activation to the digital facialmodel; and recalibrating the first facial pose data and the activationusing input user adjustments to the weighted activation.

In yet another aspect, a system for retargeting facial motion capturedata to a digital facial model is disclosed. The system includes: a FACSmodule to manage a plurality of action units; a retargeting module togenerate at least one weighted activation for the digital facial modelusing the facial motion capture data and the plurality of action units;an animation module to generate a facial animation frame by applying theat least one weighted activation to the digital facial model; and atuning interface module to generate recalibrated action units for theFACS module in accordance with input user adjustments to the facialanimation.

In a further aspect, a method of digital facial animation includes:capturing facial motion data; labeling the facial motion data;stabilizing the facial motion data; cleaning the facial motion datausing a FACS matrix; normalizing the facial motion data; retargeting thefacial motion data onto a digital facial model using the FACS matrix;and performing multidimensional tuning of the FACS matrix.

Other features and advantages of the present invention will become morereadily apparent to those of ordinary skill in the art after reviewingthe following detailed description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of the present invention, both as to its structure andoperation, may be gleaned in part by study of the accompanying drawings,in which:

FIG. 1 is a flowchart illustrating a method of animating a digitalfacial model;

FIG. 2 is a flowchart illustrating a method of recalibrating actionunits of a FACS matrix;

FIG. 3 is a functional block diagram of a system for animating a digitalfacial model;

FIG. 4 is a flowchart illustrating a method of performance driven facialanimation;

FIG. 5 is an image of actors on a motion capture set;

FIG. 6A is a three-part image depicting a neutral facial pose;

FIG. 6B is a three-part image depicting a brow lowering facial pose;

FIG. 6C is a three-part image depicting a lip corner pull facial pose;

FIG. 7A is a three-part image depicting a mouth stretch facial pose;

FIG. 7B is a three-part image depicting a lip stretch facial pose;

FIG. 8 is a three-part image depicting variability in facial motioncapture data quality;

FIG. 9 illustrates an example computation of weights for a weightedcombination of FACS poses;

FIG. 10A is an image depicting an example lip articulation for thepartially-opened mouth of an animated character;

FIG. 10B is an image depicting an example lip articulation for thefully-opened mouth of an animated character;

FIG. 10C is an image depicting an example lip articulation for theclosed mouth of an animated character;

FIG. 11 depicts an example FACS pose before and after a tuning phase;

FIG. 12 depicts an example of solved animation frames before and after atuning operation;

FIG. 13 depicts another example of solved animation frames before andafter a tuning operation;

FIG. 14A illustrates a representation of a computer system and a user;and

FIG. 14B is a functional block diagram illustrating the computer systemhosting a facial animation system.

DETAILED DESCRIPTION

Certain implementations as disclosed herein provide for systems andmethods to implement a technique for capturing motion of one or moreactors or objects. For example, one method as disclosed herein utilizesa motion capture (“MOCAP”) system to capture the body and facial motionand surfaces of multiple actors using cameras and optical markersattached to the actors. The MOCAP system builds data from the capturedimages to use in animation in a film.

Features provided in implementations include, but are not limited to,cleaning and stabilizing facial data using a Facial Action Coding System(FACS) regardless of the capturing medium, including normal low/highresolution video and MOCAP, for example; facial animation using FACS;and multidimensional tuning of FACS action units. The FACS, proposed byPaul Eckmann and Wallace Friesen, and based on a library of well-studiedfacial expression set from psychology, has been the basis of drivingcomputer graphics (CG) models.

After reading this description it will become apparent to one skilled inthe art how to practice the invention in various alternativeimplementations and alternative applications. However, although variousimplementations of the present invention will be described herein, it isunderstood that these embodiments are presented by way of example only,and not limitation. As such, this detailed description of variousalternative implementations should not be construed to limit the scopeor breadth of the present invention as set forth in the appended claims.

When an exact replica of an actor's performance is desired, manyprocesses work by tracking features on the actor's face and usinginformation derived from these tracked features to directly drive thedigital character. These features include, for example, use of a fewmarker samples, curves or contours on the face, and a deforming surfaceof the face. These processes are intended to programmatically translatedata derived from the performance of an act to animations on a digitalcomputer graphics (“CG”) face. The success of these processes oftendepends on the quality of data, the exactness and realness required inthe final animation, and facial calibration. The expertise of bothartists (trackers, facial riggers, technical animators) and softwaretechnology experts is also often required to achieve a desired endproduct. Setting up a facial processing pipeline to ultimately producehundreds of shots of many actors' performances, captured simultaneously,and requiring inputs and controls from artists and animators, presentssignificant further challenges.

A performance will be understood to be a visual capture of an actor'sface. In most instances, the actor is talking and emoting eitherindividually or in a group with other actors. This is often done bycapturing a video performance of the actor. The video frames can be usedeither purely for reference by an animator, for further processing toextract point samples, or for deforming 3-D surfaces which are thenretargeted onto a digital facial model. Various technological hurdlesmust be overcome before the 2-D or 3-D reconstructed data can be used,including calibrating cameras, tracking points, and reconstructing 3-Dinformation.

Other media types such as audio have been used to capture a voiceperformance and drive digital facial models. Most of the workapproximates the lip and mouth movement of lines of speech but does nothave explicit information relating to other areas of the face such asbrows, eyes, and the overall emotion of the character. These attributeshave to be either implicitly derived or added during post-processing. Inone implementation, facial puppeteering has been used to drive a digitalfacial model. In another implementation, a control device such as acyber glove is used to input control commands, and finger movements areretargeted onto the digital facial model.

While these forms of capture for driving a digital facial model haveyielded results, a common mode of data for driving a facial animationhas been optical data, used to reconstruct certain facial feature pointsthat are retargeted onto a digital facial model.

There are different ways in which the facial expressions can becaptured. In one implementation, the MOCAP system captures data of thebody and face together. The facial data are targeted onto an animatedcharacter whose face is stylized and does not conform to the face of theactual actor. In another implementation, the images are directed towardproducing a realistic animation on a character that is intended to lookreal, and its face to perform realistically. In a furtherimplementation, the facial MOCAP data are acquired separately in asitting position and the facial animation generated is blended inkeyframed body shots.

Making data-driven facial animation work well is a challenge becausethere are many requirements that produce varying levels of data qualityincluding the different types of systems used, the number of peoplesimultaneously captured, and the nature of facial only versus face andbody capture. The MOCAP system can support multiple approaches and socan be adapted to these, and other, various production requirements.

In one implementation, face and body motion are captured simultaneouslywith multiple cameras (e.g., 200 cameras) positioned about a “capturevolume.” An example capture volume is about 20 feet×20 feet×16 feet inlength, width, and height, respectively. Multiple infrared markers(e.g., 80 markers) are coupled to an actor's face and used to capturethe actor's performance. It will be appreciated that otherconfigurations of cameras, capture volumes, and markers can be used. Thecaptured data are reconstructed in 3-D using the positions of themultiple cameras during post-processing. A tool such as IMAGEWORKS™proprietary IMAGEMOTION™ technology, adapted to capturing and processingMOCAP data, can be used. The number of actors acting in motion capturevolume can vary from low to high numbers depending on size of thevolume, camera resolutions, strength of optical lights and signals, andother related parameters.

During a typical MOCAP session, all the actors are instructed to standapart. Each actor then individually performs a standard T-pose position,where the legs are placed together, hands are stretched out, and theface is relaxed to a neutral position. The T-pose is useful for searchand standardization purposes for both the body and face in the MOCAPdata during post-processing. Also, each MOCAP “take” ends in all theactors returning to the standard T-pose in the capture volume with theface back to the relaxed neutral position. The T-pose is used by thefacial pipeline in a normalization process to ensure that markerplacements on a second day of MOCAP performances, for example,correspond to those on the day of the calibration (also referred to asthe “master T-pose”). FIG. 5 depicts actors each performing a T-pose ina capture volume. In another instance of a motion capture adaptation(known as an ADR session), only one actor is acting in a sit downposition with sensors looking at the actors face. In such cases a T-Posewould correspond to a neutral pose of the face only with no bodyposition.

According to a Facial Action Coding System (“FACS”), the human face hasmuscles that work together in groups called “actions units.” A FACSprovides a framework for determining when certain action units aretriggered and how to assign to each action unit a relative influence ina facial pose. The FACS was initially designed for psychologists andbehavioral scientists to understand facial expressiveness and behavior,though it has also been adapted to other areas.

Facial expressions have been categorized into 72 distinct action units.Each action unit defines a muscular activity (“activation”) thatproduces a momentary change in facial appearance. These changes infacial appearance vary from person to person depending on facialanatomy, e.g., bone structure, fatty deposits, wrinkles, the shapes ofvarious facial features, and other related facial appearances. However,certain commonalities are seen between people as these action units aretriggered. An action unit used in a FACS is based on the location on theface of the facial action, and the type of facial action involved. Forexample, the upper face has muscles that affect the eyebrows, forehead,and eyelids; the lower muscles around the mouth and lips form anothergroup. Each of these muscles works in groups to form action units; andthese action units can be broken down further into left and right areasof the face, which can be triggered asymmetrically and independently ofeach other. In general, all the action units suggested by a FACS providea broad basis for dynamic facial expressions that can be used in CGanimation.

A motion capture system may use a FACS as a foundation for capturing andretargeting facial MOCAP data on an animated character's face. Prior toa MOCAP performance, each actor performs a series of calibration posesthat include extreme versions of all the action units. The reconstructed3-D facial pose data corresponding to an action unit capture the extremefacial expression used by the actor to perform that action unit. In oneimplementation, the FACS includes 64 poses, some of which are split intoleft and right positions. In another implementation, 18 phoneme posescorresponding to articulated phonemes are also included.

FIGS. 6A-6C and 7A-7B illustrate a few of the action units used in theMOCAP system based on FACS. As discussed above, FACS proposes upwards of72 action units that include expressions involving facial muscles andhead motion. FIG. 6A is a three-part image depicting a neutral facialpose; FIG. 6B is a three-part image depicting a brow lowering facialpose; FIG. 6C is a three-part image depicting a lip corner pull facialpose; FIG. 7A is a three-part image depicting a mouth stretch facialpose; and FIG. 7B is a three-part image depicting a lip stretch facialpose. In each of FIGS. 6A-6C and 7A-7B, the actual FACS reference, theactor's performance, and the retargeted expression on the character areshown from left to right.

As discussed above, in one implementation, data capture is performedusing an optical system capturing both body and face motion of one ormore actors performing in a capture space. This implementation usespassive optical components including infrared cameras to captureinfrared light reflected by the markers. An image so captured is a lowentropy image comprising mostly black areas where no infrared light issensed, and white dots representing the reflective markers. The size ofa white dot in the image varies depending on whether the dot is a bodymarker (large) or face marker (small), on the distance of the actor (andhence the marker) from the camera, and on whether any occlusions haveoccurred, where the occlusions are usually caused by the actors.

The low entropy images provide at least two advantages: (1) the camerascan capture and record images at higher definitions and at higher framerates, typically at 60 Hz; and (2) 3-D reconstruction of the capturedmarker data triangulates each marker across multiple images withdifferent viewpoints to locate the marker in space. The ability toassociate corresponding points automatically is greatly improved byusing only white dots on a black background.

After 3-D reconstruction, the markers are represented by spatialpositions (i.e., x, y, z) in a plurality of data frames. However, thedata are often noisy, do not have temporal associativity (i.e.,consistent labeling) across all of the data frames, and may have gaps.FIG. 8 is a three-part image depicting variability in facial motioncapture data quality. Shown in the left-most part of FIG. 8 is anexample of good quality data. Shown in the middle part of FIG. 8 is anexample of lower quality data. And, shown in the right-most part of FIG.8 is an example of poor quality data. These problems can be addressed ina learning-based approach taking information both from a facial datamodel and the temporal associativity of the data.

The markers reconstructed for each data frame can have both body markersand facial markers. Both the body markers and facial markers requirelabeling prior to facial data processing. That is, each marker isassigned a unique identification that persists across the data frames.Labeling all body and facial markers according to their trajectories isa cumbersome and error prone process, especially when a large number ofmarkers is visible in the volume. In one implementation, a two-stepprocess based on the size disparity between body markers (larger) andfacial markers (smaller) is used. First, 3-D reconstruction is performedwhere facial markers are ignored and only body markers are reconstructedand labeled, usually according to velocity-based constraints. Next, the3-D reconstruction is performed to acquire facial markers, but whichwill usually also capture body markers. The body markers are removed byeliminating all markers labeled in the first step, leaving only facialdata remaining. In another implementation, labeling the facial markersis automated based on a library of action units (a “FACS matrix”)specifically tailored to the corresponding actor's face.

During a performance, the actor is typically moving around in thecapture volume. The movements result in a translation of the facemarkers in accompaniment with the body while the actor is speaking andemoting. To retarget the facial marker data onto a digital facial model,it is beneficial to stabilize the facial data by nullifying thetranslational and rotational effects of body and head movements.Particular difficulties arise with respect to stabilization becausefacial markers do not necessarily undergo a rigid transform to astandard position as the actor performs. Rigid movements are caused byhead rotations and the actor's motion, but when the actor emotes andspeaks, many of the facial markers change positions away from theirrigid predictions. A few stable point correspondences are typicallysufficient to solve for an inverse transformation. However, it isfrequently difficult to determine on a frame-by-frame basis whichmarkers are relatively stable, having undergone only a rigidtransformation, and which have not been subject to other movementsrelated to emoting or speaking. Noise in the 3-D reconstructed positionsof the markers can further impede the determination of a rigidtransformation.

In one implementation, a hierarchical solution is invoked by firstperforming a global (or gross) stabilization using markers thatgenerally do not move due to facial expressions, such as markers coupledto the head, ears and the nose bone. The solution is then refined with alocal (or fine) stabilization by determining marker movements relativeto a facial surface model.

After the facial data have been stabilized, the facial data may bemissing markers due to occlusions, lack of visibility in the cameras,noise caused by errors in 3-D reconstructions, and/or mislabeledmarkers. In one implementation, a cleaning and filtering tool is usedwhich includes a learning system based on good facial model data. Thecleaning and filtering tool generates estimates of the positions ofmissing markers, removes noise, and in general ensures the viability ofall the markers. The system is scalable to handle data generated by wideranges of facial expression, and can be tuned to modify the dynamics ofthe facial data.

The cleaning tool utilizes the underlying FACS theory to organizemarkers into groups of muscles. Muscle movements can be used toprobabilistically estimate the likely positions of missing markers. Amissing marker location is estimated spatially in a neighborhood points,and estimated temporally by analyzing ranges of motion of the markers.In one implementation, a probabilistic model and a corresponding markermuscle grouping are tuned to each actor.

Once all the marker positions are determined (or estimated), standardfrequency transforms are used to remove noise in the data. It will beappreciated that high frequency content, which is normally categorizedas noise, may also represent quick, valid movements of the actor'smuscles and changes in the actor's facial expression.

When capturing a long performance, such as a movie spanning over morethan one day, actors typically remove and reattach motion capturemarkers. Although steps are taken to ensure that the markers are placedat the same positions on the face each time, small differences betweenmarker placements at the daily positions are common. These differencescan significantly affect the retargeting solutions described below.Normalization is therefore an important component of adjusting themarker placements so that the differences in the daily positions do notcompromise the extent of facial expression performed by the actor, andthe facial expression is accurately transferred onto the digital facialmodel.

In one implementation, normalization is accomplished in two steps. EachMOCAP take starts and ends with the actors performing a T-pose, asdiscussed in relation to FIG. 5. The T-pose of each actor in asubsequent MOCAP take is aligned with the master T-pose of the actordetermined during calibration. Aligning a T-pose to the master T-poserelies on the use of various relaxed landmark markers. For example, thecorners of the eyes and mouth are used because they are expected tochange very little from day to day. Offset vectors for each marker arecomputed according to discrepancies in the alignment of the T-pose andmaster T-pose. The offset vectors are applied to the T-pose of thecorresponding MOCAP take so that each marker in the T-pose isidentically aligned to the markers of the master T-pose. The offsets arepropagated through the actor's performance during that day, thusnormalizing the data in all the frames.

As discussed above, a FACS provides a set of action units or posesdeemed representative of most facial expressions. In one implementation,MOCAP frames of calibration poses performed by an actor relating tofacial expressions corresponding to FACS poses (i.e., action units) arecaptured. Some of the calibration poses are broken into left and rightsides to capture an asymmetry that the actor's face may exhibit.Subsequently, incoming frames of the actor's performance are analyzed inthe space of all the FACS poses (i.e., action units) of the FACS matrix.The action units may thus be viewed as facial basis vectors, and aweight for each is computed for an incoming data frame. A weightedcombination of action units (i.e., facial basis vectors, FACS poses) isdetermined to approximate a new pose in an incoming data frame.

FIG. 9 illustrates an example computation of weights w₁, w₂ . . . w_(n)for a weighted combination of FACS poses. Computing weights w₁, w₂ . . .w_(n) determines an influence associated with each of n FACS actionunits. In one implementation, computing the weights includes a linearoptimization. In another implementation, computing the weights includesa non-linear optimization.

The weights are applied to the associated n FACS action units togenerate a weighted activation. The weighted activation is transferredonto a digital facial model rigged with a facial muscle system.

In one implementation, the facial poses of an animated character,corresponding to FACS poses, are generated by an artist using a facialrig. In another implementation, a digital facial model setup is based onIMAGEWORKS'™ proprietary character facial system. The character facialsystem helps pull and nudge vertices of the digital facial model so thatresulting deformations are consistent with the aspects of a human face.

The digital facial model includes different fascia layers blended tocreate a final facial deformation on the digital facial model. Thefascia layers in one implementation include a muscle layer that allowsfacial muscle deformations, a jaw layer that allows jaw movement, avolume layer that control skin bulges in different facial areas, and anarticulation layer for pronounced lip movement. The muscle layerincludes skull patches with muscle controls that deform the face. Themuscle controls are activated by weighted activations generated fromMOCAP data. The jaw layer helps to control movements of the jaw of thedigital facial model. The volume layer adds volume to the deformationsoccurring on the digital facial model. It aids in modeling wrinkles andother facial deformations, which can be triggered by weightedactivations generated from MOCAP data. The articulation layer relates tothe shape of the lips as they deform. In particular, it aids incontrolling the roll and volume of lips, essential when the lips thinout or pucker in facial expressions. FIG. 10A is an image depicting anexample lip articulation for the partially-opened mouth of an animatedcharacter. FIG. 10B is an image depicting an example lip articulationfor the fully-opened mouth of an animated character. FIG. 10C is animage depicting an example lip articulation for the closed mouth of ananimated character.

The fascia layers can be constructed onto the digital facial model.Incoming MOCAP data are mapped, or retargeted, onto the digital facialmodel as weighted activations that trigger the fascia layers. Asdiscussed above, an incoming frame of MOCAP data is analyzed in thespace of all of the action units (i.e., facial basis vectors) of theFACS matrix. The resulting weights quantify the proportional influencethat each of the action units of the FACS matrix exerts in triggeringthe fascia layers. However, because the weights are obtained usingmathematical methods (e.g., linear and non-linear optimization), theresulting expression created on the digital facial model sometimes failsto replicate facial deformations naturally recognized as articulating adesired expression. That is, although the facial retargeting achievedusing the various mapping solutions may be optimally correct in amathematical sense, the resulting facial expressions may not conform tothe desired look or requirements of a finalized animation shot.

There can be several reasons for these nonconforming results. The actormay not perform according to the calibration poses provided initiallyfor the FACS matrix, thus causing the action units to benon-representative of the actor's performance; retargetinginconsistencies sometimes arise when mapping mathematically correctmarker data to an aesthetically designed face; the digital facial modelmay conform poorly to the actor's face; marker placements on the actor'sface may differ adversely from day to day; and/or the desired animationmay be inconsistent with the actions performed by the actor, such aswhen a desired expression is not present in the MOCAP data, or anexaggeration of the captured expression is attempted.

A multidimensional tuning system can use tuning feedback provided by ananimator to reduce the effects of incorrect mathematical solutions. Thisis mathematically achievable since the facial basis vectors of the FACSmatrix mimic real human expressions and can therefore be easily editedby the animator. After a FACS solve and retargeting is performed, theanimator can adjust one or more selected frames (e.g., five to tenframes having unacceptable results) to achieve a “correct look” in theanimator's artistic judgment. The adjustment is performed by modifyingthe weights resulting from the FACS solves associated with the poses inthe selected frames. The modified poses are then used to update andoptimize the FACS matrix. The updated FACS matrix thus includes actionunits based on actual marker ranges of motion as well as the modifiedweights. In one implementation, non-linear mathematical optimizationtools are used to optimize the action unit pose data and activationlevels. In the tuning process, artistic input is taken from the artistor user by modifying weights so that the overall expression suiteclosely matches the desires of a user. This is done on a few frames. Thetuning process then learns from all the changed weights resulting in anew/modified FACS matrix. The modified FACS matrix is used in subsequentsolves on the MOCAP data in order to apply the adjusted weightingprovided by the animator on the poses in the selected frames. Themodifications in the FACS library are also incorporated in the otherframes, generating improved results over the entire animation. Further,should the modified FACS library generate results that are still notsatisfactory, the animator can perform further adjustments to buildupdated FACS libraries.

FIG. 11 depicts an example of FACS poses before and after a tuningoperation. The left image of FIG. 11 shows a lip shut phoneme positionoverlaid before and after tuning. The right image of FIG. 11 shows a liptightener pose before and after tuning. The new marker positions (inblack) have been adjusted to an optimized location based on theanimator's corrected weighting values over a few tuned frames. Thischange is shown on the two poses depicted, but often occurs on moreposes depending on the nature of the animator's input adjustments.

FIG. 12 and FIG. 13 depict examples of solved animation frames beforeand after a tuning operation. In FIG. 12, the left image depicts a framesolved using the initial, calibrated FACS matrix, and the right imagedepicts the same frame solved using the modified (tuned) FACS matrix.The resulting effect is concentrated on the right lip tightener of thepose. In FIG. 13, the left image depicts a frame solved using theinitial, calibrated FACS matrix, and the right image depicts the sameframe solved using the modified (tuned) FACS matrix. The actor isuttering the beginning of the word “please.” The solve using theinitial, calibrated FACS matrix does not show the lips closed to say thefirst syllable whereas the solve using the modified FACS matrix doesshow the lips closed.

FIG. 1 is a flowchart illustrating a method 100 of animating a digitalfacial model. At 110, action units are defined for a FACS matrix. In oneimplementation, as discussed above, the FACS matrix includes 64 actionunits, each action unit defining groups of facial muscle groups workingtogether to generate a particular facial expression. Action units canfurther be broken down to represent left and right sides of the face,and thus compose asymmetrical facial poses.

The action units of the FACS matrix are calibrated, at 120. Typically,each actor has a unique, individualized FACS matrix. In oneimplementation, each action unit is calibrated by motion capturing theactor's performance of the pose corresponding to the action unit. Facialmarker data are captured as described above, FACS cleaned andstabilized, and assigned to the FACS matrix in correspondence with theparticular action unit. In another implementation, the actor performsthe pose in an extreme manner to establish expected bounds for markerexcursions when the pose is executed during a performance.

After the calibration (at 120) is completed, MOCAP data are acquiredduring a performance. New facial pose data are received one frame at atime, at 130, as the MOCAP data are generated during performance andacquisition. The frame of MOCAP data comprises volumetric (3-D) datarepresenting the facial marker positions in the capture space. In oneimplementation, the volumetric data are FACS cleaned and stabilized, asdescribed above, before being received (at 130).

Weights are determined, at 140, which characterize a weightedcombination of action units approximating the new facial pose data.Action units represent activations of certain facial muscle groups, andcan be regarded as facial basis vectors, as discussed above. As such,one or more action units—including all of the action units in the FACSmatrix—are used as components which, in a weighted combination,approximate the new facial pose data. That is, the new facial pose dataare characterized as some combination of the predefined action units inthe FACS matrix. Determining the weights involves optimally fitting aweighted combination of the facial pose data associated with each actionunit to the new facial pose data. In one implementation, a linearoptimization, such as a least squares fit, is used to compute theoptimal combination of weights. In another implementation, a non-linearoptimization is used to perform the fit.

Once the weights are determined (at 140) a weighted activation isgenerated, at 150. In one implementation, the weights are applied tomuscle group activations associated with each action unit and theresulting activations are combined to generate a weighted activation.The weighted activation is then applied to the digital facial model, at160.

If more MOCAP data frames are available for processing (determined at170), then a new frame of MOCAP data is received, at 130, and theprocess continues as described above. If no more MOCAP data frames areavailable, then the process continues by recalibrating the FACS matrix,at 180. In one implementation, recalibrating the FACS matrix (at 170) isundertaken while more MOCAP data frames are available, on command by theuser.

Recalibrating the FACS matrix (at 170) can include receiving adjustmentsto the weighted activation from the user. For example, if the userdesires a modification to a pose in a particular frame, the user mayselect the frame and adjust the weights used to generate the weightedactivation. Since the weights correspond to predefined action units, andthe action units correspond to distinct facial movements (i.e.,activations of certain facial muscle groups), the pose can be adjustedby manipulating the weights corresponding to facial muscle groupscontrolling the particular aspect of the pose intended to be changed.For example, where movement of the left corner of the mouth is definedin an action unit, the left corner of the mouth of the digital model ismoved to a more extreme position, or less extreme position, bymanipulating the weight associated with that action unit. Thus, ananimator or artist, for example, is able to control various aspects of afacial expression by manipulating natural components of the face (i.e.,action units).

FIG. 2 is a flowchart illustrating the recalibration of action units ofa FACS matrix (at 180). At 200, frames containing poses on the digitalfacial model which the user wishes to modify are selected. For example,out of thousands of data frames, five to ten frames might be selectedfor modification of the facial data. For each selected frame, theweights are modified to generate the desired facial pose, at 210. In oneimplementation, the corresponding action units are modified accordinglyto include the adjusted weights, and are exported to the FACS matrix.Thus, the FACS matrix is updated with new versions of those particularaction units, modified to accommodate the user's expectations for theparticular facial poses associated with them. In another implementation,the same data set originally processed according to the methodillustrated in FIG. 1 is reprocessed using the updated FACS matrix.While the data of the particular frames that were adjusted will now beretargeted to the digital facial model in a more desirable manner, otherfacial pose data for which the modified action units nevertheless play asignificant role in terms of weighting will also be retargeted in such away as to improve the overall quality of the animation.

FIG. 3 is a functional block diagram of a system 300 for animating adigital facial model, including a retargeting module 310, a FACS module320, an animation module 330, and a tuning interface module 340.

The retargeting module 310 receives cleaned, stabilized facial MOCAPdata, and action units from the FACS module 320. The FACS module 320receives cleaned, stabilized calibration data, and maintains a pluralityof action units in a FACS matrix, the functionality of which isdescribed above. The cleaned, stabilized calibration data are used tocalibrate the action units of the FACS matrix maintained by the FACSmodule 320. The retargeting module 310 generates a weighted activation,according to weights determined therein characterizing a weightedcombination of action units which approximates the facial pose datarepresented by the received facial MOCAP data.

The animation module 330 receives a weighted activation and generatesanimation data. The animation data include the results of activating adigital facial model according to the weighted activation. In oneimplementation, the animation module 330 maintains a digital facialmodel, and includes a rigging unit 332, which is used to generate fascialayers on the digital facial model. In particular, the fascia layers arecomponents of the digital facial model to which the weighted activationis applied to generate the animation data. In another implementation,the animation module 330 includes a transfer unit 334 which applies theweighted activation to the fascia layers of the digital facial model.

A tuning interface module 340 is configured to receive input useradjustments, and is used by a user to generate recalibrated action unitsfor the FACS matrix maintained by the FACS module 320. In oneimplementation, the tuning interface module 340 includes a frameselection unit 342 used by a user to select animation data frames inwhich the resulting pose of the digital facial model is deemedunsatisfactory. The frame selection unit 342 can be used to select anynumber of frames from the frames of animation data. In anotherimplementation, the tuning interface module 340 includes a weightmodification unit 344, which is used by the user to modify the weightscorresponding to appropriate action units for the purpose of adjusting apose of the digital facial model to achieve a desired result. Once theweights have been adjusted to the user's satisfaction, the tuninginterface module 340 conveys information regarding the adjusted actionunit to the FACS module 320, where the information is received and usedto update the FACS matrix.

FIG. 4 is a flowchart illustrating a method 400 of performance drivenfacial animation. At 410, facial motion data are captured. In oneimplementation, as discussed above, MOCAP cameras disposed about acapture space are used to capture infra-red light reflected byreflective markers coupled to an actor's body and face. The reflectedlight appears as white dots on a black background, where the white dotsrepresent the markers in the images. The images from the MOCAP camerasare used to reconstruct sequential frames of volumetric data in whichthe marker positions are located. The facial data are segmented from thevolumetric data (essentially by filtering out the body data) and arelabeled, at 420. The facial data are stabilized, as discussed above, at430. The facial data are then cleaned using a FACS matrix, at 440. Thefacial data are then normalized, at 450, to remove positional offsetdiscrepancies due to day-to-day variations in marker placement, forexample.

At 460, the facial data are retargeted frame-by-frame to a digitalfacial model using weighted combinations of action units of the FACSmatrix. A multidimensional tuning is then performed by a user, at 470,where action units comprising a pose on the digital facial model aremodified by the user to achieve a more desirable result. The modifiedaction units are incorporated into the FACS matrix as updates. Theupdated FACS matrix is then used to generate a higher quality ofanimation output.

FIG. 14A illustrates a representation of a computer system 1400 and auser 1402. The user 1402 can use the computer system 1400 to process andmanage performance driven facial animation. The computer system 1400stores and executes a facial animation system 1416, which processesfacial MOCAP data.

FIG. 14B is a functional block diagram illustrating the computer system1400 hosting the facial animation system 1416. The controller 1410 is aprogrammable processor which controls the operation of the computersystem 1400 and its components. The controller 1410 loads instructionsfrom the memory 1420 or an embedded controller memory (not shown) andexecutes these instructions to control the system. In its execution, thecontroller 1410 provides the facial animation system 1416 as a softwaresystem. Alternatively, this service can be implemented as separatecomponents in the controller 1410 or the computer system 1400.

Memory 1420 stores data temporarily for use by the other components ofthe computer system 1400. In one implementation, memory 1420 isimplemented as RAM. In another implementation, memory 1420 also includeslong-term or permanent memory, such as flash memory and/or ROM.

Storage 1430 stores data temporarily or long term for use by othercomponents of the computer system 1400, such as for storing data used bythe facial animation system 1416. In one implementation, storage 1430 isa hard disk drive.

The media device 1440 receives removable media and reads and/or writesdata to the inserted media. In one implementation, the media device 1440is an optical disc drive.

The user interface 1450 includes components for accepting user inputfrom the user of the computer system 1400 and presenting information tothe user. In one implementation, the user interface 1450 includes akeyboard, a mouse, audio speakers, and a display. The controller 1410uses input from the user to adjust the operation of the computer system1400.

The I/O interface 1460 includes one or more I/O ports to connect tocorresponding I/O devices, such as external storage or supplementaldevices (e.g., a printer or a PDA). In one implementation, the ports ofthe I/O interface 1460 include ports such as: USB ports, PCMCIA ports,serial ports, and/or parallel ports. In another implementation, the I/Ointerface 1460 includes a wireless interface for communication withexternal devices wirelessly.

The network interface 1470 includes a wired and/or wireless networkconnection, such as an RJ-45 or “Wi-Fi” interface (including, but notlimited to 802.11) supporting an Ethernet connection.

The computer system 1400 includes additional hardware and softwaretypical of computer systems (e.g., power, cooling, operating system),though these components are not specifically shown in FIG. 14B forsimplicity. In other implementations, different configurations of thecomputer system can be used (e.g., different bus or storageconfigurations or a multi-processor configuration).

It will be appreciated that the various illustrative logical blocks,modules, and methods described in connection with the above describedfigures and the implementations disclosed herein have been describedabove generally in terms of their functionality. In addition, thegrouping of functions within a module or subunit is for ease ofdescription. Specific functions or steps can be moved from one module orsubunit to another without departing from the invention.

One implementation includes one or more programmable processors andcorresponding computer system components to store and execute computerinstructions, such as to provide the various subsystems of a motioncapture system (e.g., calibration, matrix building, cleanup,stabilization, normalization, retargeting, and tuning using FACStechniques).

Additional variations and implementations are also possible. Forexample, the animation supported by the motion capture system could beused for film, television, advertising, online or offline computercontent (e.g., web advertising or computer help systems), video games,computer games, or any other animated computer graphics videoapplication. In another example, different types of motion capturetechniques and markers can be used, such as optical markers other thaninfrared, active optical (e.g., LED), radio (e.g., RFID), paint,accelerometers, deformation measurement, etc. In another example, acombination of artistic input and mathematical processes is used tomodel a face which is activated using retargeting solutions. In afurther example, mathematical, heuristic, and aesthetically based rulesare developed to enhance the fidelity of muscle and skin movements onthe digital facial model when the animated character talks.

The above description of the disclosed implementations is provided toenable any person skilled in the art to make or use the invention.Various modifications to these implementations will be readily apparentto those skilled in the art, and the generic principles described hereincan be applied to other implementations without departing from thespirit or scope of the invention. Thus, it will be understood that thedescription and drawings presented herein represent implementations ofthe invention and are therefore representative of the subject matterwhich is broadly contemplated by the present invention. It will befurther understood that the scope of the present invention fullyencompasses other implementations that may become obvious to thoseskilled in the art and that the scope of the present invention isaccordingly limited by nothing other than the appended claims.

1. A method of animating a digital facial model, the method comprising:defining a plurality of action units; calibrating each action unit ofsaid plurality of action units via an actor's performance; capturingfirst facial pose data; determining a plurality of weights, each weightof said plurality of weights uniquely corresponding to said each actionunit, said plurality of weights characterizing a weighted combination ofsaid plurality of action units, said weighted combination approximatingsaid first facial pose data; and recalibrating at least one action unitof said plurality of action units using input user adjustments.
 2. Themethod of claim 1, wherein said each action unit includes a secondfacial pose data and an activation.
 3. The method of claim 2, whereinsaid calibrating each action unit includes calibrating said secondfacial pose data of said each action unit using calibration pose dataderived from a calibration performance corresponding with said eachaction unit.
 4. The method of claim 3, further comprising cleaning andstabilizing said calibration pose data.
 5. The method of claim 2,wherein said weighted combination includes a weighted combination ofsaid second facial pose data of said each action unit.
 6. The method ofclaim 5, wherein said determining a plurality of weights includes anoptimization of a correspondence between said first facial pose data andsaid weighted combination of said second facial pose data.
 7. The methodof claim 6, wherein said optimization includes a linear optimization. 8.The method of claim 7, wherein said linear optimization includes aleast-squares method.
 9. The method of claim 6, wherein saidoptimization includes a non-linear optimization.
 10. (canceled)
 11. Themethod of claim 2, wherein said recalibrating at least one action unitincludes recalibrating said second facial pose data.
 12. The method ofclaim 2, wherein said recalibrating at least one action unit includesrecalibrating said activation.
 13. The method of claim 2, wherein saidactivation of said each action unit is directed to a fascia layer. 14.The method of claim 13, wherein said fascia layer includes a musclelayer.
 15. The method of claim 13, wherein said fascia layer includes ajaw layer.
 16. The method of claim 13, wherein said fascia layerincludes a volume layer.
 17. The method of claim 13, wherein said fascialayer includes an articulation layer.
 18. The method of claim 1, whereinsaid plurality of action units comprises a FACS matrix.
 19. The methodof claim 1, further comprising cleaning and stabilizing said firstfacial pose data.
 20. A method of animating a digital facial model, themethod comprising: defining a plurality of action units, each actionunit of including first facial pose data and an activation; calibratingsaid first facial pose data using calibration pose data derived from aplurality of captured calibration performances, each calibrationperformance of said plurality of captured calibration performancescorresponding with said each action unit; deriving second facial posedata from another calibration performance of said plurality of capturedcalibration performances; determining a plurality of weights, eachweight of said plurality of weights uniquely corresponding to said eachaction unit, said plurality of weights characterizing a weightedcombination of said facial pose data, said weighted combinationapproximating said second facial pose data; and recalibrating said firstfacial pose data and said activation using input user adjustments.
 21. Asystem for retargeting facial motion capture data to a digital facialmodel, the system comprising: a FACS module to manage a plurality ofaction units; a calibration module to calibrate each action unit of theplurality of action units via an actor's performance; and a tuninginterface module to generate recalibrated action units for said FACSmodule in accordance with input user adjustments to a facial animationframe.
 22. The system of claim 21, wherein said animation moduleincludes a rigging unit to generate said digital facial model.
 23. Thesystem of claim 22, wherein said rigging unit generates at least onefascia layer on said digital facial model.
 24. The system of claim 23,wherein said animation module includes a transfer module to apply saidat least one weighted activation to said at least one fascia layer. 25.The system of claim 21, wherein said tuning interface module includes aframe selection unit to select said facial animation frame for tuning.26. (canceled)
 27. A method of digital facial animation, the methodcomprising: defining a plurality of action units in a FACS matrix;calibrating each action unit of said plurality of action units via anactor's performance; capturing facial motion data; labeling said facialmotion data; stabilizing said facial motion data; cleaning said facialmotion data using said FACS matrix; normalizing said facial motion data;retargeting said facial motion data onto a digital facial model usingsaid FACS matrix; and recalibrating using multidimensional tuning ofsaid FACS matrix.
 28. The method of claim 27, wherein multidimensionaltuning uses tuning feedback provided by an animator to reduce theeffects of incorrect mathematical solutions in the FACS matrixassociated with poses in selected frames.
 29. The method of claim 28,wherein the tuning feedback is performed by modifying weights resultingfrom the solutions in the FACS matrix associated with the poses in theselected frames.
 30. The method of claim 29, wherein the modifiedweights are used to update and optimize the FACS matrix, resulting inthe FACS matrix including action units based on actual marker ranges ofmotion as well as the modified weights.